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(54) Queued arbitration mechanism for data processing system 



(57) A queued arbitration mechanism transfers all 
queued processor bus requests to a centralized system 



controller/arbiter in a descriptive and pipelined manner. 
Transferring these descriptive and pipelined bus re- 
quests to the system controller allows the system con- 
troller to optimize the system bus utilization via prioriti- 
zation of all of the requested bus operations and pipelin- 
ing appropriate bus grants. Intelligent bus request infor- 
mation is transferred to the system controller via encod- 
ing and serialization techniques. 
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Dedcription 

CROSS REFERENCE TO RELATED APPUCATiONS 

5 ihts application for pat nt is r lated to the f tlowing applications for pat nt filed concun^entty her with: 

EFFICIENT ADDRESS TRANSFER TECHNIQUE FOR A DATA PROCESSING SYSTEM, serial number 
08/AAAAAA (HQ9-94-032); 

10 DUAL LATENCY STATUS AND COHERENCY REPORHNG FOR A MULTIPROCESSING SYSTEM, serial number 

08/BBB.BBB (HQ9-94-033); 

SYSTEM AND METHOD FOR DETERMINING SOURCE OF DATA IN A SYSTEM WITH INTERVENING CACHES, 
serial nunriberOa/CCC.CCC (HQ9-94-034); 

IS 

METHOD AND APPARATUS FOR REMOTE RETRY IN A DATA PROCESSING SYSTEM, serial number 
Oa/DDD.DDD (AT9-94-163); 

ARRAY CLOCKING METHOD AND APPARATUS FOR INPUT/OUTPUT SUBSYSTEMS, serial number 
20 0a/EEE.EEE(AT9-94-164); 

DATA PROCESSING SYSTEM HAVING DEMAND BASED WRITE THROUGH CACHE WITH ENFORCED 
ORDERING, serial number OS/FFRFFF (AT9-94-157): 

2S COHERENCY AND SYNCHRONIZATION MECHANISMS FOR I/O CHANNEL CONTROLLERS IN A DATA 

PROCESSING SYSTEM, serial number OS/GGG.GGG (AT9-94-162); 

ALTERNATING DATA VAUD CONTROL SIGNALS FOR HIGH PERFORMANCE DATA TRANSFER, serial number 
Oa/HHH,HHH (HQ9-94^35); 

30 

LOW LATENCY ERROR REPORTING FOR HIGH PERFORMANCE BUS. serial number 08/111,111 (HQ9-94-036). 

Each of such cross-referenced applications are hereby incorporated by reference into this Application as though 
fully set forth herein. 

TECHNICAL FIELD OF THE INVENTION 

The present invention relates in general to data processing systems and, in particular, to a system and method for 
intelligent communication of bus requests and bus grants within a data processing system. 

40 

BACKGROUND OF THE INVENTION 

Conventional data processing systems, especially multiprocessor systems, allocate access to the shared system 
bus coupling the various bus devices to system memory through a mechanism whereby individual bus devices each 

45 control access to the system bus. Typically, each bus device will queue if s individual bus requests for various operations 
internally. Then, each bus device makes the determination of which of the various operations it wishes to perform on 
the system bus by sending the appropriate corresponding bus request to the system controller. Thus, each individual 
bus device determines internally which of its bus requests has higher priority. The system controller is then required to 
arbitrate between the received bus requests from the individual bus devices. 

so One disadvantage of this arbitration mechanism is that a portion of the decteion process for accessing the various 
resources coupled to the system bus is delegated to each of the bus devices. As a result, the system controller is only 
able to view a portion of all of the various requests from the Individual bus devk^es. since each of the individual bus 
devices retains arxJ queues a significant numt>er of bus requests. Thus, there is a need in the art for a nrKxe efficient 
arbitration mechanism for granting access to the system bus. 

55 

SUMMARY OF THE INVENTION 

It is an object of the present invention to centraliz the decision-rmking process for granting access to the system 
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bus. In an attainment of this object, the present invention provides a nnechanism f transferring all of the qu ued bus 
requests from th individual bus devices to the system controller, which has a centralized knowledge of th availability 
of aD of the system resources coupled to the system bus. 

Th system controil r samples th bus devices' requests on a cycle-by-cycle basis. The requests ar encoded. 
s which allows each of th bus devices to precisely conomunicate to the system controller each of their internally 'qu ued' 
operations. Quickly transfening these 'descriptive and pipelined' bus requests from each of the bus devices to a cen- 
tralized control point, alkwvs the system controller to 'optimize* the system bus utilization by prioritizing all of the re- 
quested bus operations and pipelining the appropriate bus grants. 

One advantage of the present invention is that it provides an ability to transfer 'intelligent' bus request information 
10 from each bus devk:e to the system controller, artd provkles the ability to transfer multiple packets of bus requests 
infonmatkxi (via encoding and serialization technkjues). 

Another advantage of the present invention is that the bus requests are compact and can be issued in a pipelined 
manner and that bus grants may be pipelined to either the same bus devk^ or different bus devces. 

Yet anc^er advantage of the present lnventk)n is that it supports latch-to-latch or non-latch-to-latch implementatkxis. 
IS Those skilled in the art will appreciate the benefit of accommodating both implementatbns. (Latdi-to-latch implemen- 
tatnns albw higher system bus ckx:k rates, while non-latch-to-latch implementatbns provides k>wer system bus laten- 
cies.) 

Yet stilt another advantage of the present invention is that the queuing of descriptive bus requests albws the system 

controller to efficiently control, distribute, and allocate system bus resources. 
20 And, yet still arx>ther advantage the present invention is that the system controller may resolve system level 

multiprocessor problems such as deadkxks and livekx:ks. Unlike tracfitional arbitration technk^ues, the present tnventbn 

bus does not require bus devces to adhere to any arbitratkxi fairness' protocols. 

Another advantage of the present inventbn is that the bus devk^es may support speculative bus requests and the 

system controller nr^y support speculative bus grants. 
2S The foregoing has outlined rather broadly the features and technical advantages of the present inventk>n in order 

that the detailed description of the inventk)n that follows nrtay be better understood. Additiona] features and advantages 

of the inventkvi will t>e described hereinafter which form the subject of the claims of the inventk>n. 

BRIEF DESCRIPTION OF THE DRAWING 

30 

For a more complete understanding of the present invention, and the advantages thereof, reference is now made 
to the following descriptkxis taken in conjunction with the accompanying drawings, in whk^h: 

FIGURE 1 illustrates a block cSagram of a data processing system in accordance with the present inventk>n; 

FIGURE 2 illustrates a block c£agram of the system controller illustrated in FIGURE 1 ; and 

FIGURE 3 illustrates an exemplary protocol for granting bus grants for bus requests from one of the bus devk^es 
illustrated in FIGURE 1. 

40 

DETAILED DESCRIPTION OF A PREFERRED Ef^BODIMENT OF THE INVENTION 

With the foregoing hardware in mind, it is possible to explain the process-related features of the present inventbn. 
To rrwre clearty describe these features di the present inventkm, discusskxi of other conventbnal features is omitted 

45 as being apparent to those skilled in the art It is assumed that those skilled in the art are familiar with a multiuser, 
multiprocessor operating system, smd in partk:ular with the requirements of such an operating system for memory man- 
agement including virtual memory, processor scheduling, synchronizatkxi facilities for both processes and processors, 
message passing, ordinary device drivers, terminal and network support, system (nitializatk>n, interrupt management, 
system call facilities, and administrative f^ilities. 

so Referring now to FIGURE 1 , a data processing system whch advantageously embodies the present inventkxi will 

be described. Multiprocessor system 1CX) includes a number of processing units 102, 104. 106 operatively connected 
to a system bus 1 08. Also connected to the system bus 1 08 is a merrx^ry controller 1 1 0, which controls access to system 
memory 112, and I/O channel controllers 114, 116, and 118. Additkxially, a high performance IVO devce 120 may be 
connected to the system bus 1 08. Each of the system elements described 1 02-1 20. inclusive, operate under the control 

ss of system controller 1 30 whch communk:ates with each unit connected to the system bus 108 by point to point lines 
such as 1^ t processor 102. 134 1 processor 104, 136 to processor 106, 140 to memory controil r 110, 144 to I/O 
channel controller 11 4, 146 to I/O channel controller 11 6. 148tol/Ochann Icontroll r 118. and 150to high performance 
I/O devk:e 120. Requests and grants of bus access are all controlled by system controil r 1 30. 
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IVO channel controller 114 controls and is connected t system I/O subsystem and nativ I/O subsyst m 1 60. 

Each processor unit 102, 104. 106 may indud a processor and a each storage device. 

The present invention rray be implemented with a clock synchronous system bus and separate address and data 
buses. Furthermor . as illustrated in FIG URE 1, bus requests and bus grants ar transferred, in a pre! red emtxxJiment. 
point-to-point (unicfirectkanally r bidirectionally) between th bus devices and system controller 1 30. This unidirectional 
point-to-point topdogy is capable of supporting totally independent and concun^ent address and data bus requests and 
address and data bus grants. /Vdditionally. it provides for system scalability without affecting the request-to-grant speed, 
and is capable of supporting a 'private' communication protocol between the various bus devices and system controller 
130. 

Some of the possible encoded bus requests may be as follows: 
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Typical Bus Operation 


Store/Push 
Retried Store. 
Cast Out, Speculative 
Store 

Load^Address Only 
Retried Load/Address 
Only. 

Speculative 
Load/Address Only 
Load Repty/lnterventlon 
NULL 

Store/Push 
LoacUAddress Only 
Load Reply/Intervention 
CANCEL ALL 
REQUESTS 


Request Priority 


High 
Low 

High 
Low 

High 

High 
High 
High 


Cancel Queued 
Requests 


CO CO CD CO 

O O OOmmcDm 


Queued Request 


S S Sooooo 


Data Bus Request 


o> _ <0_co_co_ 

^ z ?z:?z>!iz 


Address Bus 
Request 


m CD OOcDOOO 

:S > z z >- z z 


Code 


(A) 
(B) 

(C) 
NULL 

(D) 

(F) 

CNCL 


a: 
m 
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O T- Ot-O'-O^ 


-ABR 
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XBR 


o O O O T- ^ 1- ^ 
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Passible bus grants may be encoded as f llows: 



-ABQ 


•DBG 


CODE 


Description 


Grant to which Bus 


Typical Bus Operation 










Requests 




0 


0 


(AD) 


Address and Data Bus 


AorD 


Store or Push or Cast Out 








Grant 






0 


1 


(AO) 


Address Onty Bus Grant 


BorE 


Load or Address Onty 


1 


0 


(DO) 


Data Onty Bus Grant 


CorF 


Load Repty or Intervention 


1 


1 


(NG) 


No Grant 







In the above tables, XBR represents a control bit, while -ABR and -DBR represent address bus requests and data 
bus requests, respectively. -ABG and -DBG represent address bus grant and data bus grant, respectively. As may be 
noted in the above bus request table, a '1 ' in the XBR portion of the bus request code represents that a particular bus 
IS device is sending a request that is not to be queued and which cancels all previously queued requests from that particular 
bus device. 

Note thai ttietemis 'Cast Out". "Store", "Push", "Load". "Address Only", "Load Reply". "Intervention", "Speculative 
Load", "Speculative Store", "Retried Store", "Load Reply", "Retried Load" are all terms for operations well-known in the 
art an6 are to be interpreted according to their traditional references In the art 
20 Referring next to FIGURE 2, ttiere is Illustrated a block diagram of system controller 1 30. System controller 1 30, as 

previously illustrated in FIGURE 1, is coupled to the various bus devices via lines 132. 134. 136. 140, 144. 146, 148 
and 150. These lines carry the encoded bus requests arKi transmit the encoded bus grant information to and from the 
bus devices. 

In the folbwing discussion, reference will only be made to input latch 201 . decoder 203, reset request latches 205 

2S and request latches 207-209. which are coupled to processor 1 02 via connedon 1 32. Components 202. 204, 206, 21 0. 
211 , 212 operate in a similar manner, and may be coupled to I/O channel controller 118 via connectkxi 148. 

When a bus device, such as processor 1 02 sends a bus request to system controller 1 30 via line 1 32, it is received 
by input latch 201 . l-alches 201 . 202, 21 6 and 21 7 assist in implementing system controller 1 30 with the bus devices in 
system 100 within a latch-to-latch implementatk)n. 

30 As bus requests are sent from processor 102 to system controller 130, they are latched into input latch 201 and 

decoded by decoder 203. If the bus requests are to be queued requests, they are then latched in succession into request 
latches 207-209. In a preferred embodiment of the present invention, system controller 1 30 implements a 3-deep queue. 
Of course, system controller 1 30 could be designed by one skilled in the art to implement various otiier N-deep queues, 
where N > 0. Note that at the same time that processor 102 is sending queued requests to system controller 130 as 

3S described above, other bus devbes, such as I/O channel controller 118 may also be sending bus requests, whether 
queued or not queued, to be latched into request latches 210-212. 

Queued request prbritization logic 21 3 then observes all latched requests from all bus devices via latches 207-21 2, 
and prnritizes their requests to determine whch are to be given bus grants first For example, by reference to the bus 
request table above, high priority requests will be granted access to system bus 108 before low priority requests. Fur- 

40 thermore, logic 21 3 may be designed for a particular system 1 00 to grant bus 1 08 for a toad bus request before a store 
bus request One skilled in the art may easily implement any desired priorities for determining which requests are to 
receive granting df bus 108 and in what order for implementatk)n within logic 21 3. 

As k^ic 213 determines whk:h queued request to grant the bus to next, it then signals bus grant generatton bgk; 
215 of whksh encoded grant to generate and to which bus devfce. 

^ If decoder 203 receives one of the bus requests from the bus request table that requires previously queued requests 

to be cancelled, decoder 203 will signal reset request latch 205, whk:h resets request latches 207-209, car^^elling pre- 
vious requests from processor 1 02. Decoder 203 also sends these unqueued requests to unqueued request prioritization 
logic 214, whk^ also periorms a prioritization process between the unqueued requests received by logic 214. Logic 
214, upon determining whk:h of the unqueued requests is to be granted access to s^tem bus 108 next, signals togk: 

so 215 of such a decision. Again, one skilled in the art may easily irr^lement any desired priorities for determining whk^h 
unqueued requests are to receive granting of bus 1 08. Logk: 21 5 receives a prioritized queued request and a prioritized 
unqueued request, and determines whk:h of these is to be granted access to system bus 108 next Generally, since 
unqueued requests have a high priority, they will be granted access to system bus 108 before queued requests. 

Bus grant generatton togic 215 generates the encoded bus grants illustrated in the table above. These bus grants 

ss are latched out of system controller 1 30 by output latches 21 6-21 7. Thus, if togc 21 3 determines that a queued request 
from processor 1 02 is to receive the next bus grant, bus grant gen ration logic 21 5 will produce the appropriate encoded 
grant, whk^h will be latched from latch 216 to processor 102, whch will tti n utiliz syst m bus 108 in th requested 
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nnannen 

Also, the encoded grant from bus grant generation bgtc 21 5 will b used by reset request latc^tes 205 to reset th 
appropriate queued request latch 207, 208, or 209. 

R ferring next to FIGURE 3, th re is illustrated an xemplary protocol of bus requests and bus grants for one of 
5 th bus devices, such as processor 102. The bus requests ar pipelined and sent from processor 102 via lin 132 to 
system controller 130 as indicated. Through the aboveKlescribed process, system controller produces the pipeline of 
bus grants as indicated in FIGURE 3. Note the numerous "no grants' ("NGs*) within the pipeline of grants, which may 
indicate thai one or more of the other bus devices in system 1 00 is currently being sen/ed by system controller 1 30. The 
'null' notations incficate that processor 102 is currently not transmitting a bus request. 
10 Since a queued bus request may have previously been transmitted by processor 102, a 'nuir request does not 
imply that processor 102 does not need access to the system bus 108. 

The example shows that processor 102 is first in need of a store or push bus operation (noted by code A) artd is 
next in need of a load or address only operation (noted by code B). At some time later, system controller 1 30 grants the 
address and data bus for the code A requested operation and then later grants the address bus only in response to the 
t5 code B request. 

The example bus request pipeline also indicates a serialization technique whereby two consecutive encoded re- 
quests from a particubr bus device indicate to system controller 1 30 that the bus request is a bw priorfty request. Such 
a low priority request may be in response to a previously "retried* bus operation from one of the other bus devices. 
Retries on the system bus 108 often result in prolonged livelocks and maybe even a deadlock. A deadlock may be 

20 defined as an infinite livekx:k. A livekxk may be defined as a condition on the system bus 108 in which a bus device 
'A' retries an operatbn by bus device 'B' and bus device 'B' retries an operatkvi by bus device 'A' and this cydk^al 
pattern continues until another condition 'alters' this pattern. Uvekxk conditions are well known in the art. Livekx^k 
conditions severely degrade system performance due to the inefficient usage of the system bus resources. Therefore, 
it is advantageous to differentiate a bus request from a prevk>usty retried bus request Furthermore, retried bus requests 

25 often get retried again due to 'busy' system resources. Thus, it is also advantageous to have these retried bus requests 
contain a k>w priority in order to more ^ciently utilize the system bus resources. It may further be preferable, then, to 
grarrt access to the bus for bw prbrity requests in a randomized fashion and high prbrity requests in a prioritized fashbn. 
The random generatbn of grant to bw prbrity requests avobs the cyclbal system bus retries, thus avobing livebcks 
and deadbcks. The prioritized generation of grant to high arKi low priority requests effbiently realizes the system bus 

30 bandwkith. 

As indicated within FIGURE 3, the bus request encoded as a bad reply or interventbn, is given a data only bus 
grarrt before the address and data bus grant in response to the low prbrity requests (errcoded with an A). This illustrates 
how system controller 1 30 granted access to bus 108 to a higher prbrity request instead of a bw prkurty request. 

Also illustrated is how the second bus request B is cancelled by bus request D, which as indicated in the bus request 
35 table is not intended to be a queued request, and which informs system controller 130 to cancel all prevbus requests 
from processor 102. Since bus request D is nc^ queued inside controller 130, bus request D remains active until it 
receives a grant. Such a situatbn is decoded by decoder 203 and transferred to unqueued request prbritization bgb 
214, which informs bus grant generatbn logb 215 of the unqueued request. Furthermore, decoder 203 informs reset 
request latches 205 to cancel all previous requests within the queued request latches 207-209. 
40 In the above bus request and bus grant tables, interventbn refers to a situatbn where another bus device has 

snooped a bus request and has determined that it contains within its internal cache a 'dirty.* or modified versbn of the 
requested data. A mechanism is then set in motbn whereby the requesting bus devbe is informed that data is to be 
received from the other bus device instead of system memory. A further discussion of 'inten^entkxi* is supplied within 
cross-referenced United States Patent Applbatbn No. (HQ9-94-034), whbh is hereby incorporated by reference herein. 

Note that the XBR bus request signal need not be implemented in all systems. For low cost systems, the XBR 
informatbn may be configured to a specific value in system controller 130 for certain bus devbes. 

Note that there is capacity for other encoded requests and grants via serializatbn technk^ues or the ^itbn of bus 
request and bus grant signals. Furthermore, other types of requests and other protocols may be designed into the system 
of the present inventbn as desired. 
50 Although the present inventbn and its advantages have been described in detail, it shoub be underwood that 

various changes, substitutbns and alteratbns can be made herein without departing from the spirit and scc^e of the 
invention as defined by the appended claims. 

ss Claims 

1 . A computffl^ system, comprising: 
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a plurality of bus devices including one rmor processors and one or more storage d vices; 
a system controller; 

a bus architectur coupling said plurality of bus devices and said system controller; 

first means adaptable for transfening each address and data bus request from said plurality of bus devices to 
said system controller as said each address and data bus request is generated by said plurality of bus devices; 
and 

second means adaptable for transfening. from said system controller to said plurality of bus devices, responses 
to said each address and data bus request. 

The computer system as recited in claim 1 . wherein said each address and data bus request is queued within said 
system controller and is not queued in said plurality of bus devices, and wherein said first means adaptable for 
transfening further comprises: 

means for pipelining bus requests from each one of said plurality of bus devices; and 

wherein said second means adaptable for transferring further comprises: 

means for pipelining bus grants to each one of said plurality of bus devices. 

The computer system as recited in claim 1 , wherein said second means adaptable for transferring further comprises: 

means for reordering bus grants, characterized in that at least one of said each address and data bus request 
is a speculative bus request; and 

at least one of said responses to said each address and data bus request is a speculative bus grant 

The computer system as recited in claim 1 . wherein said each address and data bus request and said responses 
to said each address and data bus request are performed on a point-to-point basis between each of said plurality 
of bus devices and said system controller; and 

said bus architecture including separate address and data buses and said storage device comprising a memory 
controller coupled to a memory device; 

said system controller sampling said each address and data bus request every bus cycle. 

The computer system as recited in claim 4, wherein one of said each address and data bus request cancels all 
previous bus requests from one d said plurality of bus devices generating said one of said each address and data 
bus request cancelling said all previous bus requests. 

The computer system as recited in claim 4, wherein an issue of one of said each address and data bus request on 
at least two consecutive bus cycles is treated as a low priority bus request by said system controller, wherein said 
system controller may issue a bus grant in response to a bus request received subsequent to said issue of said one 
of saki each address and data bus request on at least two consecutive bus cycles before issuing a bus grant in 
response to said issue of said one of said each address and data bus request on at least two consecutive bus cycles. 

The computer system as recited in daim 1 . wherein sakJ responses are ordered by said system controller according 
to priority levels of said each address and data bus request and according to an availability of system resources; arKi 
wherein one or wore of said one or rTX>re processors are virtual processors. 

A method of arbitrating bus access in a multiprocesscx system, said method comprising the steps ot 

sampling bus requests from a plurality of bus devices within said multiprocessor system in a mann r that relieves 
qu uing said bus requests in each of said plurality of bus devbes; 
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responding to said bus requests from said plurality of bus devices. 

9. Th method as recited in claim 8, wherein sad sampling is p rfomoedev ry bus cycle by a system controller coupled 
in a point-to-point mann rt said plurality of bus devices, furth r comprising th st pof: 

5 queuing said sampled bus requests in said system controller. 

10. The method as recited in claim 9. further comprising the step of: 

issuing said bus requests by said plurality of bus devices, said issuing step occurring before said sampling step 
10 for each of said bus requests; and 

wherein said issuing step further comprises the step of: 

encoding one of said bus requests to be a request either of a store operation, a bad operation, load reply, 
retried store, bus grant for said load operation, bus grant for said operation. 

IS 
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